The brain is a difficult organ to study. Unlike the heart or stomach, it determines how it perceives itself. A major challenge of neuropsychology is separating fact from intuitive, and often incorrect, conjecture. Mathematics, however, provides a rigorous and testable framework for understanding neurological processes. In August of 2014, David and Lucile Packard Professor of Computer Science and Biomedical Engineering Steven Zucker and his research group made several key contributions to the theory behind how visual information is processed by the brain.
Scientists have long struggled with the “shape-from-shading” puzzle. When the light from a 3D surface enters the eye, a 2D image is cast on the retina. The brain must then construct a 3D representation of that surface using, among other cues, the pattern of shading depicted in the 2D image. As with any mathematical theory in biology, existing models do not provide a complete picture of this process. Nevertheless, finding a better solution is important. Artists, filmmakers, and even camouflage developers have long taken advantage of the phenomenon.
Zucker and his student, Benjamin Kunsberg, developed a mathematical model that better reflects biology. A basic problem for the visual system is how to do this without knowing the direction of the light cast onto a surface. Previously existing models, however, assumed the light sources were known. Zucker’s group developed light-independent equations to model changes in shading within closed patches of an object’s surface. These changes are called “shading flows.” As Zucker explains, “The mathematical question for brains to solve is, ‘Given one of these patches of shading flow, what possible surface could give rise to it?’” Zucker’s solution solves for the orientation and three-dimensional shape of each patch based upon shading flow, and then stitches these patches back together to form a complete picture. The effective light source is an emergent one.
Even shading, however, does not tell a complete story about how the brain perceives shape. Surfaces are rarely smooth, and humans are capable of seeing many colors. It turns out that when these additional sources of information are taken into account, contradictions and ambiguities arise. Nevertheless, Zucker explains that texture and color can also be modeled as “flows,” and therefore computed similarly. A texture flow can be defined by measuring changes between regions that are “isotropic.” In an isotropic region, textural elements align in the same direction. For example, newly combed hair forms an isotropic region. The role of color can be similarly inferred by measuring changes between points to yield “isohue,” or single hue, contours. These are analogous to the isophotes, or regions of similar shade, defined through the shading flow.
A given shape may appear different depending on whether it is shaded or textured. When both elements are present, the brain has to decide which cues to follow. Zucker and Kunsberg’s solution predicts regions of interest (ROIs) within shapes where shading or texture flows serve as strong indicators of shape. Zucker and colleagues conducted a study in which people were exposed to images of 3D objects that were shaded, textured, or both. They found that participant performance correlated to ROIs predicted mathematically. Furthermore, Zucker and Kunsberg found that shading and texture complemented one another in areas where their ROIs overlapped.
It is intuitive that shading and texture are complementary in regions where they are both significant. However, this is not the case when shading and color flows run parallel to one another. Consider the tradition of college students painting campus landmarks. Imagine a rock, for example, were painted so that the flow of color matched the flow of shading on its surface as seen from a particular side. From that perspective, the boulder would appear to lose its 3D quality. It would look as if it were a peculiarly painted cardboard cutout.
In the example of our oddly painted boulder, Zucker and Daniel Holtmann-Rice think that the brain is being tricked into incorrectly distinguishing an object’s shape from the material of which its made. As Zucker explains, “We think that your brain is assuming that if the color and light flows are changing in the same direction then it’s an attribute of material.” If color and shading flows don’t align, your brain ‘knows’ that the latter can be interpreted as 3D geometry. Zucker and Holtmann-Rice are constructing stimuli for psychological studies.