Humans have the largest brain of any primate. While it seems logical to assume that overall size is very important for generating complex behaviours, brain size relative to body size has been considered to be a major factor in predicting overall brain capacity. It turns out, however, that the absolute number of neurons in the cerebral cortex, regardless of body mass, may be a more relevant factor. Here we review the ways in which brains have increased in size, why absolute brain size is sometimes important, and why the size of the human brain allowed us to have cognitive abilities that exceed those of other primates. We suggest that cognitive functions are largely mediated by the neocortex, and because the human brain scales like a typical primate brain, the large neocortex of humans contains more neurons than any other mammal, even those with larger brains such as elephants. Further, as neurons in primary sensory cortex increase in numbers with brain size at a greater rate than the increase in the number of neurons in thalamic relay nuclei, primates with larger brains and more neocortex also have more neurons to analyze these sensory inputs. As numbers of neurons increase, individual neurons are free to specialize in different ways, generating increasing variability in cell size, shape, dendritic arborization and other features. In addition, an expanded cortical sheet contains more cortical areas, thereby increasing the number of computational levels involved in information processing, decision-making, and information storage. Having more cortical areas allows any given area to become more specialized in terms of laminar and sub-laminar organization, modular organization, connectivity and function. Increases in cortical field number also allow for greater variation in the sizes of areas, and thereby different types of functional specializations. Finally, large brains have more areas that are removed from primary sensory inputs and capable of hemispheric specialization. Of course, the costs of a large brain are considerable in terms of gestation time, postnatal vulnerability, and metabolic costs. Thus, it is not surprising that most mammals have relatively small brains that are constrained in their processing capacity, but are more metabolically efficient, and mature rapidly allowing for early reproduction.