Abstract
We explore the visual world through saccadic eye movements, but saccades also present a challenge to visual processing by shifting externally stable objects from one retinal location to another. The brain could solve this problem in two ways: by overwriting preceding input and starting afresh with each fixation or by maintaining a representation of presaccadic visual features in working memory and updating it with new information from the remapped location. Crucially, when multiple objects are present in a scene the planning of eye movements profoundly affects the precision of their working memory representations, transferring limited memory resources from fixation toward the saccade target. Here we show that when humans make saccades, it results in an update of not just the precision of representations but also their contents. When multiple item colors are shifted imperceptibly during a saccade the perceived colors are found to fall between presaccadic and postsaccadic values, with the weight given to each input varying continuously with item location, and fixed relative to saccade parameters. Increasing sensory uncertainty, by adding color noise, biases updating toward the more reliable input, which is consistent with an optimal integration of presaccadic working memory with a postsaccadic updating signal. We recover this update signal and show it to be tightly focused on the vicinity of the saccade target. These results reveal how the nervous system accumulates detailed visual information from multiple views of the same object or scene.
SIGNIFICANCE STATEMENT This study examines the consequences of saccadic eye movements for the internal representation of visual objects. A saccade shifts the image of a stable visual object from one part of the retina to another. We show that visual representations are built up over these different views of the same object, by combining information obtained before and after each saccade. The weights given to presaccadic and postsaccadic information are determined by the relative reliability of each input. This provides evidence that the visual system combines inputs over time in a statistically optimal way.
