Megan D. Neilson’s research while affiliated with UNSW Sydney and other places

Planning a multi-joint minimum-effort coordinated human movement to achieve a visual goal is computationally difficult: (i) The number of anatomical elemental movements of the human body greatly exceeds the number of degrees of freedom specified by visual goals; and (ii) the mass–inertia mechanical load about each elemental movement varies not only with the posture of the body but also with the mechanical interactions between the body and the environment. Given these complications, the amount of nonlinear dynamical computation needed to plan visually-guided movement is far too large for it to be carried out within the reaction time needed to initiate an appropriate response. Consequently, we propose that, as part of motor and visual development, starting with bootstrapping by fetal and neonatal pattern-generator movements and continuing adaptively from infancy to adulthood, most of the computation is carried out in advance and stored in a motor association memory network. From there it can be quickly retrieved by a selection process that provides the appropriate movement synergy compatible with the particular visual goal. We use theorems of Riemannian geometry to describe the large amount of nonlinear dynamical data that have to be pre-computed and stored for retrieval. Based on that geometry, we argue that the logical mathematical sequence for the acquisition of these data parallels the natural development of visually- guided human movement.

At every point p on a smooth n-manifold M there exist n+1 skew-symmetric tensor spaces spanning differential r-forms ω with r=0,1,⋯,n. Because d∘d is always zero where d is the exterior differential, it follows that every exact r-form (i.e., ω=dλ where λ is an r−1-form) is closed (i.e., dω=0) but not every closed r-form is exact. This implies the existence of a third type of differential r-form that is closed but not exact. Such forms are called harmonic forms. Every smooth n-manifold has an underlying topological structure. Many different possible topological structures exist. What distinguishes one topological structure from another is the number of holes of various dimensions it possesses. De Rham’s theory of differential forms relates the presence of r-dimensional holes in the underlying topology of a smooth n-manifold M to the presence of harmonic r-form fields on the smooth manifold. A large amount of theory is required to understand de Rham’s theorem. In this paper we summarize the differential geometry that links holes in the underlying topology of a smooth manifold with harmonic fields on the manifold. We explore the application of de Rham’s theory to (i) visual, (ii) mechanical, (iii) electrical and (iv) fluid flow systems. In particular, we consider harmonic flow fields in the intracellular aqueous solution of biological cells and we propose, on mathematical grounds, a possible role of harmonic flow fields in the folding of protein polypeptide chains.

This discussion paper supplements our two theoretical contributions previously published in this journal on the geometric nature of visual space. We first show here how our Riemannian formulation explains the recent experimental finding (published in this special issue on size constancy) that, contrary to conclusions from past work, vergence does not affect perceived size. We then turn to afterimage experiments connected to that work. Beginning with the Taylor illusion, we explore how our proposed Riemannian visual–somatosensory–hippocampal association memory network accounts in the following way for perceptions that occur when afterimages are viewed in conjunction with body movement. The Riemannian metric incorporated in the association memory network accurately emulates the warping of 3D visual space that is intrinsically introduced by the eye. The network thus accurately anticipates the change in size of retinal images of objects with a change in Euclidean distance between the egocentre and the object. An object will only be perceived to change in size when there is a difference between the actual size of its image on the retina and the anticipated size of that image provided by the network. This provides a central mechanism for size constancy. If the retinal image is the afterimage of a body part, typically a hand, and that hand moves relative to the egocentre, the afterimage remains constant but the proprioceptive signals change to give the new hand position. When the network gives the anticipated size of the hand at its new position this no longer matches the fixed afterimage, hence a size-change illusion occurs.

Bringing together a Riemannian geometry account of visual space with a complementary account of human movement synergies we present a neurally-feasible computational formulation of visuomotor task performance. This cohesive geometric theory addresses inherent nonlinear complications underlying the match between a visual goal and an optimal action to achieve that goal: (i) the warped geometry of visual space causes the position, size, outline, curvature, velocity and acceleration of images to change with changes in the place and orientation of the head, (ii) the relationship between head place and body posture is ill-defined, and (iii) mass-inertia loads on muscles vary with body configuration and affect the planning of minimum-effort movement. We describe a partitioned visuospatial memory consisting of the warped posture-and-place-encoded images of the environment, including images of visible body parts. We depict synergies as low-dimensional submanifolds embedded in the warped posture-and-place manifold of the body. A task-appropriate synergy corresponds to a submanifold containing those postures and places that match the posture-and-place-encoded visual images that encompass the required visual goal. We set out a reinforcement learning process that tunes an error-reducing association memory network to minimize any mismatch, thereby coupling visual goals with compatible movement synergies. A simulation of a two-degrees-of-freedom arm illustrates that, despite warping of both visual space and posture space, there exists a smooth one-to-one and onto invertible mapping between vision and proprioception.

We present a Riemannian geometry theory to examine the systematically warped geometry of perceived visual space attributable to the size–distance relationship of retinal images associated with the optics of the human eye. Starting with the notion of a vector field of retinal image features over cortical hypercolumns endowed with a metric compatible with that size–distance relationship, we use Riemannian geometry to construct a place-encoded theory of spatial representation within the human visual system. The theory draws on the concepts of geodesic spray fields, covariant derivatives, geodesics, Christoffel symbols, curvature tensors, vector bundles and fibre bundles to produce a neurally-feasible geometric theory of visuospatial memory. The characteristics of perceived 3D visual space are examined by means of a series of simulations around the egocentre. Perceptions of size and shape are elucidated by the geometry as are the removal of occlusions and the generation of 3D images of objects. Predictions of the theory are compared with experimental observations in the literature. We hold that the variety of reported geometries is accounted for by cognitive perturbations of the invariant physically-determined geometry derived here. When combined with previous description of the Riemannian geometry of human movement this work promises to account for the non-linear dynamical invertible visual-proprioceptive maps and selection of task-compatible movement synergies required for the planning and execution of visuomotor tasks.

Adaptive Model Theory is a computational theory of the brain processes that control purposive coordinated human movement. It sets out a feedforward-feedback optimal control system that employs both forward and inverse adaptive models of (i) muscles and their reflex systems, (ii) biomechanical loads on muscles, and (iii) the external world with which the body interacts. From a computational perspective, formation of these adaptive models presents a major challenge. All three systems are high dimensional, multiple input, multiple output, redundant, time-varying, nonlinear and dynamic. The use of Volterra or Wiener kernel modelling is prohibited because the resulting huge number of parameters is not feasible in a neural implementation. Nevertheless, it is well demonstrated behaviourally that the nervous system does form adaptive models of these systems that are memorized, selected and switched according to task. Adaptive Model Theory describes biologically realistic processes using neural adaptive filters that provide solutions to the above modelling challenges. In so doing we seek to model the supreme modeller that is the human brain.

Recently Latash, Scholz, and Schöner (2007) proposed a new view of motor synergies which stresses the idea that the nervous system does not seek a unique solution to eliminate redundant degrees of freedom but rather uses redundant sets of elemental variables that each correct for errors in the other to achieve a performance goal. This is an attractive concept because the resulting flexibility in the synergy also provides for performance stability. But although Latash et al. construe this concept as the consequence of a "neural organization" they do not say what that may be, nor how it comes about. Adaptive model theory (AMT) is a computational theory developed in our laboratory to account for observed sensory-motor behavior. It gives a detailed account, in terms of biologically feasible neural adaptive filters, of the formation of motor synergies and control of synergistic movements. This account is amplified here to show specifically how the processes within the AMT computational framework lead directly to the flexibility/stability ratios of Latash et al. (2007). Accordingly, we show that quantitative analyses of experimental data, based on the uncontrolled manifold method, do not and indeed cannot refute the possibility that the nervous system tries to find a unique (optimal) solution to eliminate redundant degrees of freedom. We show that the desirable interplay between flexibility and stability demonstrated by uncontrolled manifold analysis can be equally well achieved by a system that forms and deploys optimized motor synergies, as in AMT.